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Abstract 

In this technical report, we describe the development of cognitive diagnostic test items that form 
the basis of the diagnostic system for Project DIVIDE (Dynamic Instruction Via Individually 
Designed Environments). The construct underlying the diagnostic test is division of fractions. 

We include a description of the process we used to identify the cognitive attributes for division 
of fractions. We present the hierarchical association between attributes that lead up to mastery of 
the domain. Finally, we describe item writing procedures and evidence for technical adequacy of 
the resulting item bank. 
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Introduction 

The purpose of project DIVIDE (Dynamic Instruction Via Individually Designed 
Environments) in mathematics is to develop an integrated cognitive diagnostic assessment and 
instructional delivery system. Applying the design flexibility and analytic power of recent 
developments in computer technology, this system will provide individualized instruction in 
mathematical computation skills and support conceptual understanding to students studying 
division of fractions. This system is designed to support students with disabilities in the general 
education setting. 

A component of this project is the development of a computer-based cognitive diagnostic 
assessment system. The results from administering this test are used to classify students into the 
instructional system that most closely aligns with their mastery of the prerequisite knowledge. 
The purpose of this technical report is to describe the development of the cognitive diagnostic 
test items. Included in this description is the process used to identify the cognitive attributes 
underlying division of fractions. We present the hierarchical association of skills and knowledge 
associated with mastery of division of fractions. Additionally, we describe item writing 
procedures and steps taken to evaluate the technical adequacy of the items. 

Identification of Cognitive Attributes 

In this project, we assigned students to instructional modules based on results from the 
cognitive diagnostic assessment. Accurate classification of students into instructional modules 
requires carefully designed assessments that align with the cognitive model underlying 
performance in the domain. The cognitive model is composed of attributes that are domain 
specific prerequisite skills and knowledge needed to demonstrate mastery in the targeted task 
(Chipman, Nichols, & Brennan, 1995; Leighton & Gierl, 2007; Tatsuoka & Tatsuoka, 1986). 
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Attributes are typically isolated through careful task analyses, expert review, verbal protocols, 
and other inquiry methods for analyzing student thinking processes (Gorin, 2006). 

Task analysis. To begin this project, the research team conducted a task analysis of the 
skills and knowledge needed to divide fractions (see Appendix A for biographies of the research 
team). First, we identified the types of division of fractions problems. Using the general model of 
division of fractions, 17 + d ^ (where a, b, c, d, e, f are whole numbers; c ^ 0 and f ^ 0 ), division 

of fractions problems can be classified into three general categories: a) a proper fraction is 
divided by a proper fraction (e.g., * + where a - C ami U - 0 ), b) a fraction is divided by a whole 

number (e.g., a + g, where a = o ar.i s = □), and c) problems involving dividing mixed numbers (e.g., 

■r 

Sr-t- 4 , st-h £7 where anOordnO). 

Second, the research team consulted mathematics textbooks and discussed the process for 
dividing fractions with a mathematician to determine the precise steps involved in this procedure. 
The mathematical rationale can be summarized: 



M + N = X (N =0) M = XN (N = 0) (1) 
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at ad 

fc + d"b X a 

The attributes emerged by examining the mathematical rationale and isolating the specific steps 
needed for students to understand and execute each step of the procedure. 

Expert review. To verify the attributes, members of the research team presented the 
mathematical rationale and proposed attributes to two mathematics education experts (for 
biographies of the Mathematics Education Expert and Mathematics Instructional Design Expert 
see Appendix A). The purpose of this review was to refine the attribute list by removing 
irrelevant attributes or adding necessary skills that were missed during the task analysis. The 
review included a description of the project, review of the vocabulary associated with cognitive 
diagnosis, and a discussion of the proposed attribute structure. We presented the mathematical 
rationale for division of fractions, discussed the conceptual and procedural knowledge needed to 
solve division of fractions problems, and solved several sample problems using various 
approaches students might try. 

As a result of this review, the original set of attributes was modified. Four attributes were 
omitted because the skills and knowledge were identified as unnecessary to master the 
procedures or concepts of division of fractions. Four attributes were added to encompass the 
range of skills and knowledge students needed to master division of fractions. The revised list of 
attributes is provided in Appendix B. 

We conducted verbal protocols with students to collect additional evidence about the 
cognitive model (see Ketterlin-Geller and Liu, in preparation). 

Association Between Attributes 



To determine the hierarchical structure of skills and knowledge needed to solve division 
of fractions problems, the research team evaluated the association among attributes. Articulating 
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these relations is an essential step for determining the necessary item types that must be sampled 
in the cognitive diagnostic test. From the administration of these items, it should be clear which 
attributes students have and have not mastered. As such, the relation across attributes can 
significantly influence the resulting instructional decisions. 

To determine these relations, the research team carefully analyzed the sequence of skills 
and knowledge needed to divide fractions. Each attribute was evaluated to determine the 
knowledge and skills that were required to master the range of examples within that attribute. We 
considered sample problems for each attribute that included whole numbers as well as rational 
numbers. 

To avoid confirmation bias, each member of the research team independently created a 
diagram specifying the association among attributes. Then, the research team met to discuss the 
diagrams. Because mastery of the subsequent attributes requires that students understand, 
recognize, and define fractions, each member of the research team identified Definition of 
Fractions as the prerequisite skill for division of fractions. The remaining associations are 
specified in Appendix C. 

Item Writing 

Instrument Development 

The cognitive diagnostic test is designed for computer delivery in a group setting. Members 
of the research team along with a Mathematics Content Expert (see Appendix A for biography) 
wrote a total of 252 items. Multiple-choice items were created for efficiency in the computer 
delivery. Each item had three distractors and one correct answer. Distractors were carefully 
written to reflect potential errors in student thinking related to the attribute. Errors included 
computational errors, conceptual errors, procedural errors, and strategy errors. The computer 
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randomly assigned the location of the correct answer in the list of possible answers. Scoring was 
dichotomous. 

The content of the test encompasses the attributes for division of fractions. Items included a 
range of problems from computational to situated within a context. Some items were designed to 
elicit conceptual understanding or required reasoning skills by the student. Graphics were used in 
instances where they explained the problem, provided a visual clue to clarify the context, or were 
integral to the stem or answer choices. Irrelevant graphics were not included. The reading level 
of the items was intentionally constrained to the 5 th grade level, however, readability statistics 
were not calculated for each item. Whenever possible, plain language and simple, straight- 
forward statements were incorporated into the items. 

Content-Related Evidence for Validity 

Internal review. Members of the research team conducted an internal review of the items 
to verify the content and alignment with the attributes. Each item was reviewed for the 
appropriateness of the content, graphics, and language. Additionally, the reviewers assigned an 
attribute(s) to the item. These classifications were used to verify the attribute assignment for each 
item. At least two internal reviewers provided feedback for each item. According to the 
reviewers’ comments, the original item writer modified the items. 

Teacher review. Eight teachers from local public schools reviewed the items. All 
participating teachers taught mathematics at public schools from 2 years to 22 years, with an 
average of 12 years. Four teachers earned Masters degrees in mathematics or education. Three 
teachers had bachelor degrees in mathematics. 

Teachers analyzed each item for grade-level appropriateness in terms of understandability 
of language and vocabulary, content or concepts, graphics, potential bias in language and/or 
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content, clarity of directions and answers, and effectiveness of distractors. The test items and 
distractors were rated on a scale of 1 to 4 for each criterion. A rating of 1 meant that the 
item/distractors were not at all appropriate based on the criterion (or very biased); a rating of 2 
meant that the item/distractors were somewhat appropriate based on the criterion (or somewhat 
biased); rating of 3 meant that the item/distractors were appropriate based on the criterion (or not 
biased); and a rating of 4 meant that the item/distractors were extremely appropriate based on the 
criterion (or not biased and has multi-cultural components to it). In instances where the teachers 
provided a rating of 3 or lower, they were asked to provide additional suggestions and comments 
regarding the measures. 

For 112 items, teachers pointed out possible problems regarding grade-level 
appropriateness. Twenty items were noted as having language or vocabulary that was too 
complex for 5 th grade students; simplified text was suggested. Twenty-two items were identified 
as having complex concepts. The graphics for 6 items were considered inappropriate. Bias was 
identified for two items. Clarity and effectiveness of the distractors was questioned for 52 items. 
For these items, teachers provided alternate distractor suggestions. The research team reviewed 
all suggestions and made revisions based on teacher feedback. 

Math expert review. Five graduate students in mathematics at the University of Oregon 
reviewed all items. All reviewers were enrolled in the Ph.D. program in mathematics and had 
earned Bachelors degrees in mathematics. Three students had completed a Masters degree in 
mathematics or physics. All participants had previous experience teaching mathematics to 
college students. 

The math experts were asked to identify the attribute(s) measured by each item and to 
evaluate the adequacy of measurement and effectiveness of distractors for each item. The 
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specific considerations are as follows. Items and distractors were evaluated on a 4-point scale for 
each criterion. A rating of 1 meant that the item did not measure the attribute identified, and the 
distractors were effective; a rating of 2 meant that the item somewhat measured the attribute 
identified, and the distractors were somewhat effective; rating of 3 meant that the item mostly 
measured the attribute identified, and the distractors were mostly effective; and a rating of 4 
meant the item effectively measured the attribute identified, and the distractors were extremely 
effective. If they give the item a score of 1 or 2 on a category, they were encouraged to provide a 
suggestion. 

The attribute classifications by the math experts differed from those made by the research 
team for 86 items. In most of these items, the math experts suggested that the items were 
measuring additional attributes that had not previously been assigned to the item. For example, in 
most cases where the item measured the relationship between multiplication and division, the 
math experts also suggested that these items were measuring whether or not students could 
balance an equation. Based on reviewer feedback, attribute classifications were changed for 47 
items when the suggestion was justifiable. 

Additionally, the math experts suggested revising the language for 17 items to increase 
the mathematical precision of the concepts presented, clarity of the distractors, and/or 
appropriateness of the graphics. The research team reviewed the wording suggestions and 
modified the language according to suggestions. 

Pilot Test 

Subjects. Six hundred four students in grades 5-7 participated in the pilot study. Most 
participants lived in Oregon; however, students from Washington, Missouri, Idaho, and 
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Massachusetts also participated. Roughly equal numbers of female (52%) and male (48%) 
students participated. 

Procedures. A convenient sample of teachers was sent an email inviting their students to 
participate in a computer-based test on division of fractions. Approximately 24 teachers agreed 
to participate. Once enrolled in the testing session, students were randomly assigned to take an 
18-item subtest. Each subtest included 5 anchor items and 13 unique items that were 
purposefully selected to represent a range of attributes at varying difficulty levels. The computer 
program provided all relevant instructions. Teachers proctored the testing session using 
standardized administration procedures specified in the testing manual. Fidelity of 
implementation was not monitored due to the distributed nature of the participants. 

Results. Results were analyzed using a 1 -parameter item response theory (IRT) model. Of 
the 252 items, 13 items under-fit and 4 items over-fit the model based on mean square residual fit 
statistics. The average item fit was 1.00 (SD=0.32). The item calibrations are reliably estimated 
with a reliability of .85. Item difficulty ranged from -3.99 to 3.65 on the logit scale. The 
item/person map (see Appendix D) indicates that the calibration sample is well targeted with the 
items and the items provide a wide measure. 

Conclusions 

The purpose of this technical report was to describe the procedures used to develop the 
cognitive diagnostic test items. We described the process used to define the cognitive model 
underlying division of fractions. In addition, we discussed item writing efforts as well as 
procedures for evaluating content-related evidence for validity. Pilot test data were presented 
indicating that the items represent a range of difficulty levels and appropriately fit the IRT 
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model. The evidence presented in this technical report indicates that the cognitive model as well 
as the diagnostic test items are appropriate for the purposes of Project DIVIDE. 
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Appendix A: Biographies of Researchers and Consultants 
Principal Investigator 

The Principal Investigator has been the Director of Research Projects with Behavioral 
Research and Teaching at the University of Oregon since 2003. Additionally she has served as an 
assistant professor with Educational Leadership since 2005. She holds a Ph. D. in Educational 
Leadership with an emphasis on applied measurement. She earned her teaching credentials for 
secondary science as well as K-12 administrative licensure. She is also the co-principal 
investigator for a research project examining the alignment of content standards in mathematics 
with IEP goals and how skills and perceptions influence goals. She has been working as a key 
researcher on projects investigating features of universally designed mathematics assessments 
and research on effects of accommodations on measurement of mathematics performance. 
Research Team Member 1 

The Research Team Member holds a Bachelor of Science and has been a research 
assistant and project coordinator for federally funded grants and state contracts for 5 years. He 
assisted in the creation of a web-delivered math assessment researching effectiveness of 
accommodations. Additionally, he has been involved in the development of the alternate 
assessment in mathematics for students with significant cognitive disabilities at both the 
elementary and high school levels for 4 years. The Research Team Member also was a team 
member of an item writing team for creating math items for math screening tests and assisted in 
development of accommodated versions of math items. He has also been a part of several 
research teams conducting multi-state research projects examining comparability of math 
performance on different alternate assessments. He is completing a Master’s degree in Special 
Education and has taken coursework in mathematics instruction and assessment design. 
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Research Team Member 2 

The second Research Team Member has a B. S. in Mathematics as well as a B.A. and a 
M.A in Special Education. She is currently pursuing a Ph. D. in Special Education at the 
University of Oregon. She has experiences in developing and validating Mathematics 
Curriculum-Based Measures (CBM) for grades 2 through 4 in her master’s program. In addition, 
she created math computation items for students in grades 3 to 8 for a project examining 
information sources for Individualized Education Program (IEP) decision making. She also 
revised alternate assessment items in mathematics for significant cognitive disabilities at both the 
elementary and secondary level. She has a secondary special educator license specialized in 
mathematics. She taught students with significant cognitive disabilities, hearing impaired, 
blindness, and autism at the elementary, middle, and high school levels as a teacher and a 
volunteer for several years in Korea. 

Research Team Member 3 

The third member of the research team completed his Ph.D. in Educational Psychology 
with an emphasis on psychometric theory, statistical decision analysis, and behavioral research 
methods. The Research Team Member was a special education teacher in New York State and 
completed a Masters degree in Special Education. He has remained active in the field as a senior 
research associate, associate professor, and consultant to government and research organizations. 
Current research interests include item response theory (IRT) modeling in the context of 
assessment for special populations, and the specification of cut scores for optimal decision 
making. The Research Team Member presents regularly at national conferences, addressing 
scale construction, scale performance standards, and bias in measurement systems designed for 
diverse populations. 
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Mathematics Education Expert 

The Mathematics Education Expert was an associate professor of the department of 
Special Education and Clinical Sciences at the University of Oregon for 7 years and is currently 
serving as a Dean of the School of Education and Human Development at the Southern 
Methodist University. He has a teaching license of mathematics in California and Michigan. He 
is a member of National Council of Teachers of Mathematics and American Mathematical 
Association. He has diverse experiences in math teaching as a classroom teacher for several 
years in California and an associate training director at U. S. Peace Corps in Lesotho, Southern 
Africa. He worked as a principal investigator of Early Mathematics Project from 2004 to 2007. 
Mathematics Instructional Design Expert 

The Mathematics Instructional Design Expert, M. A., has been a Senior Instructor and a 
Research Assistant in the Department of Special Education at the University of Oregon for over 
20 years. She taught for twelve years with students with learning disabilities, behavior disorders, 
deaf-blind, and autism at both elementary and middle school levels. She taught math to students 
in early math preskills through middle school math. Additionally, she has taught special 
education math instructional methods for about 15 years at the University of Oregon. She is a 
supervisor for practicum students and student teachers in implementing math instruction in 
special education settings. She also participated in writing a K math curriculum. She holds an 
Oregon teaching license in Elementary education (K-8) with endorsements in special education 
(K-12) and hearing impairment (K-12). 

Mathematics Content Expert 

The Mathematics Content Expert is working as a high school math teacher as well as a math 
coach at Willamette High School. She has been a high school math teacher (all grades 9 to 12) 
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for ten years. The Content Expert has a B. S. in Mathematics with minor in Economics. She also 
holds a M. A. in Economics and a MAT in Advanced Mathematics. She is a committee of 2007 
Review Benchmark cut scores for Oregon State Assessment at Oregon Department of Education. 
Additionally, she will work for 2008 Mathematics State Adopted Instructional Material Review 
as a committee. The Content Expert is a member of NCTM (National Council of Teachers of 
Mathematics). 
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Appendix B: Attribute List 

Understand that an equation should be balanced so that if you do something to one 
side, you have to do it to the other 

Understand inverse relationship between multiplication and division 
Understand the limits of division (cannot divide by a zero because you can’t have 
equal parts of nothing) 

Understand the distributive property 
Understand the definition of a fraction 

A. Relationship between two numbers; ration/proportion 

B. Sometimes includes all elements in a set 
Understand and apply inverse property of multiplication 

A. Any fraction multiplied by its inverse, equals 1 

o h 

B. Example: - x - equals 1 

b a 

Know and apply the procedures for converting the mixed number to an improper 
fraction (and converting an improper fraction to a mixed number) when a ** Q or d ^ 0. 

Apply algorithm of invert and multiply 

A. Invert the second fraction 

B. Change the sign to a times sign 

C. Multiply 

Understand definition of division of fractions 

A. “m” divided by “n” = how many copies or parts of “n” are there in “m” 

B. Any dividend (n) divided by a number between 0-1 results in a larger number 
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than the dividend (n) 

Understand definition of multiplication of fractions 

A. Divide the whole into parts 

B. Take the parts of interest 

C. “m/n” multiplied by “1/k” = divide “1/k” into “n” equal parts and take “m” of 

them 

D. Any factor multiplied by a number between 0-1 results in a smaller number 

than the factor 

Understand and apply identity property of multiplication 
A. Multiplying anything by 1, results in itself 
* Note: attributes are not in order 
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Appendix D: Item/Person Map 
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